Skip to content

[fix](parquet)fix parquet write timestamp int96 type. (2/2)#61832

Open
hubgeter wants to merge 1 commit intoapache:masterfrom
hubgeter:arrow_int96_fix_2
Open

[fix](parquet)fix parquet write timestamp int96 type. (2/2)#61832
hubgeter wants to merge 1 commit intoapache:masterfrom
hubgeter:arrow_int96_fix_2

Conversation

@hubgeter
Copy link
Copy Markdown
Contributor

@hubgeter hubgeter commented Mar 27, 2026

What problem does this PR solve?

Problem Summary:
Part 2 of pr #61760.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Mar 27, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Copy Markdown
Contributor

morningman commented Mar 28, 2026

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Mar 28, 2026

TPC-H: Total hot run time: 26433 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f6e8499a6ec528cf52ebf3ede8825b04dd7ac3f, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17651	4451	4258	4258
q2	q3	10655	758	512	512
q4	4674	344	245	245
q5	7565	1199	1037	1037
q6	172	181	143	143
q7	778	840	674	674
q8	9548	1461	1299	1299
q9	5271	4740	4685	4685
q10	6330	1923	1654	1654
q11	453	260	244	244
q12	732	569	449	449
q13	18042	2668	1935	1935
q14	227	240	212	212
q15	q16	723	734	665	665
q17	725	838	437	437
q18	5878	5435	5271	5271
q19	1123	995	607	607
q20	545	495	365	365
q21	4612	1834	1381	1381
q22	341	468	360	360
Total cold run time: 96045 ms
Total hot run time: 26433 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4835	4607	4609	4607
q2	q3	3985	4417	3851	3851
q4	874	1202	769	769
q5	4065	4404	4351	4351
q6	180	171	146	146
q7	1788	1679	1537	1537
q8	2481	2737	2566	2566
q9	7706	7391	7311	7311
q10	3781	4014	3624	3624
q11	509	449	420	420
q12	499	623	478	478
q13	2514	2981	2362	2362
q14	305	308	295	295
q15	q16	735	767	739	739
q17	1168	1359	1479	1359
q18	7316	6745	6659	6659
q19	885	878	921	878
q20	2101	2177	1977	1977
q21	3888	3539	3345	3345
q22	426	418	393	393
Total cold run time: 50041 ms
Total hot run time: 47667 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Mar 28, 2026

TPC-DS: Total hot run time: 169273 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f6e8499a6ec528cf52ebf3ede8825b04dd7ac3f, data reload: false

query5	4328	629	504	504
query6	346	223	224	223
query7	4229	469	270	270
query8	350	244	233	233
query9	8727	2764	2720	2720
query10	532	390	342	342
query11	7013	5118	4895	4895
query12	190	126	124	124
query13	1276	446	367	367
query14	5803	3754	3427	3427
query14_1	2852	2825	2858	2825
query15	202	194	178	178
query16	988	478	448	448
query17	899	737	631	631
query18	2458	450	361	361
query19	215	213	190	190
query20	139	132	128	128
query21	215	133	109	109
query22	13373	14026	14284	14026
query23	16800	16293	16437	16293
query23_1	16265	15940	15709	15709
query24	7163	1652	1248	1248
query24_1	1240	1230	1245	1230
query25	635	459	397	397
query26	1233	259	151	151
query27	2796	474	297	297
query28	4497	1840	1848	1840
query29	852	559	480	480
query30	296	229	191	191
query31	1015	937	857	857
query32	86	73	68	68
query33	510	336	301	301
query34	878	869	543	543
query35	639	681	601	601
query36	1117	1135	998	998
query37	131	98	82	82
query38	2975	2967	2906	2906
query39	850	829	821	821
query39_1	821	822	806	806
query40	234	166	134	134
query41	64	60	59	59
query42	265	264	268	264
query43	245	257	225	225
query44	
query45	199	196	183	183
query46	881	1009	632	632
query47	2863	2121	2039	2039
query48	327	327	246	246
query49	676	477	395	395
query50	737	284	223	223
query51	4123	4124	4023	4023
query52	259	261	259	259
query53	287	340	280	280
query54	316	267	258	258
query55	90	93	86	86
query56	307	320	296	296
query57	1947	1686	1596	1596
query58	286	274	265	265
query59	2777	2933	2713	2713
query60	340	340	322	322
query61	163	149	150	149
query62	618	585	534	534
query63	312	270	273	270
query64	5111	1279	1041	1041
query65	
query66	1450	453	346	346
query67	24261	24259	24216	24216
query68	
query69	398	313	287	287
query70	916	937	879	879
query71	339	305	295	295
query72	3011	2754	2472	2472
query73	533	547	317	317
query74	9666	9559	9414	9414
query75	2880	2737	2455	2455
query76	2284	1040	677	677
query77	357	377	311	311
query78	10911	11238	10520	10520
query79	2994	766	578	578
query80	1775	623	558	558
query81	568	259	229	229
query82	1009	147	120	120
query83	335	264	246	246
query84	304	126	102	102
query85	937	508	455	455
query86	481	311	331	311
query87	3160	3121	3010	3010
query88	3595	2668	2662	2662
query89	424	373	338	338
query90	2060	177	175	175
query91	173	165	142	142
query92	89	72	72	72
query93	1490	849	496	496
query94	670	327	304	304
query95	582	343	320	320
query96	644	509	225	225
query97	2481	2498	2446	2446
query98	244	223	225	223
query99	1023	988	908	908
Total cold run time: 254953 ms
Total hot run time: 169273 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Mar 28, 2026

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.89% (19935/37691)
Line Coverage 36.41% (186793/512985)
Region Coverage 32.67% (144837/443373)
Branch Coverage 33.87% (63513/187536)

@morningman
Copy link
Copy Markdown
Contributor

morningman commented Mar 28, 2026

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Mar 28, 2026

TPC-H: Total hot run time: 26742 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f6e8499a6ec528cf52ebf3ede8825b04dd7ac3f, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17659	4591	4295	4295
q2	q3	10641	769	519	519
q4	4671	350	249	249
q5	7554	1223	1011	1011
q6	176	174	145	145
q7	778	869	675	675
q8	9508	1478	1328	1328
q9	5309	4740	4729	4729
q10	6328	1912	1666	1666
q11	453	253	232	232
q12	754	565	470	470
q13	18030	2670	1961	1961
q14	235	237	216	216
q15	q16	722	747	673	673
q17	731	853	462	462
q18	6049	5377	5238	5238
q19	1390	968	626	626
q20	539	485	376	376
q21	4654	1842	1584	1584
q22	417	389	287	287
Total cold run time: 96598 ms
Total hot run time: 26742 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4838	4687	4718	4687
q2	q3	3850	4348	3794	3794
q4	900	1216	804	804
q5	4061	4383	4375	4375
q6	192	176	144	144
q7	1763	1615	1539	1539
q8	2495	2696	2651	2651
q9	7555	7426	7576	7426
q10	3752	3968	3572	3572
q11	511	441	416	416
q12	513	615	467	467
q13	2508	3076	2098	2098
q14	312	316	287	287
q15	q16	736	770	728	728
q17	1185	1299	1359	1299
q18	7284	6798	6813	6798
q19	980	969	921	921
q20	2094	2227	2011	2011
q21	3979	3476	3339	3339
q22	470	420	374	374
Total cold run time: 49978 ms
Total hot run time: 47730 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Mar 28, 2026

TPC-DS: Total hot run time: 169553 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f6e8499a6ec528cf52ebf3ede8825b04dd7ac3f, data reload: false

query5	4324	628	515	515
query6	329	231	207	207
query7	4205	470	263	263
query8	337	249	226	226
query9	8689	2732	2728	2728
query10	509	390	340	340
query11	7012	5113	4895	4895
query12	181	132	123	123
query13	1289	462	355	355
query14	5779	3739	3498	3498
query14_1	2900	2865	2870	2865
query15	207	196	176	176
query16	1007	489	462	462
query17	1124	741	638	638
query18	2463	463	361	361
query19	215	209	195	195
query20	135	129	127	127
query21	212	137	109	109
query22	13256	14215	14357	14215
query23	16969	16218	16161	16161
query23_1	16386	16022	15635	15635
query24	7358	1654	1231	1231
query24_1	1248	1266	1248	1248
query25	613	457	411	411
query26	1238	262	167	167
query27	2761	477	304	304
query28	4479	1861	1852	1852
query29	823	569	482	482
query30	299	222	192	192
query31	1017	944	893	893
query32	84	77	68	68
query33	523	338	297	297
query34	898	880	520	520
query35	646	694	605	605
query36	1066	1076	961	961
query37	134	95	85	85
query38	3007	2899	2844	2844
query39	866	830	818	818
query39_1	811	801	797	797
query40	229	148	136	136
query41	75	68	64	64
query42	262	267	261	261
query43	252	247	264	247
query44	
query45	192	188	179	179
query46	888	995	624	624
query47	3177	2108	2045	2045
query48	310	322	227	227
query49	628	454	385	385
query50	699	306	208	208
query51	4049	4031	3999	3999
query52	264	263	254	254
query53	291	334	292	292
query54	303	270	268	268
query55	92	96	85	85
query56	309	326	336	326
query57	1919	1692	1675	1675
query58	283	273	261	261
query59	2798	2952	2761	2761
query60	331	333	331	331
query61	158	153	155	153
query62	654	578	532	532
query63	309	281	278	278
query64	4993	1300	1034	1034
query65	
query66	1465	465	350	350
query67	24284	24247	24177	24177
query68	
query69	397	315	295	295
query70	978	991	942	942
query71	332	311	315	311
query72	2963	2720	2488	2488
query73	568	564	321	321
query74	9591	9549	9411	9411
query75	2860	2768	2496	2496
query76	2292	1060	699	699
query77	364	393	312	312
query78	10931	11055	10466	10466
query79	2744	808	597	597
query80	1771	628	555	555
query81	549	259	225	225
query82	1000	157	117	117
query83	330	263	249	249
query84	258	118	100	100
query85	920	513	464	464
query86	412	302	290	290
query87	3167	3116	3035	3035
query88	3628	2677	2663	2663
query89	433	379	350	350
query90	2007	184	175	175
query91	170	170	143	143
query92	80	72	74	72
query93	1176	868	509	509
query94	635	319	300	300
query95	598	404	316	316
query96	648	526	230	230
query97	2441	2486	2418	2418
query98	239	219	217	217
query99	984	980	908	908
Total cold run time: 254645 ms
Total hot run time: 169553 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Mar 28, 2026

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.89% (19935/37691)
Line Coverage 36.41% (186799/512985)
Region Coverage 32.66% (144785/443373)
Branch Coverage 33.87% (63516/187536)

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Mar 28, 2026

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.51% (27133/36913)
Line Coverage 56.99% (291443/511437)
Region Coverage 54.24% (242708/447495)
Branch Coverage 55.98% (105299/188102)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 28, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 28, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 28, 2026

PR approved by anyone and no changes requested.

@hubgeter hubgeter mentioned this pull request Mar 28, 2026
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Mar 29, 2026
@hubgeter
branch-4.1:[fix](parquet)fix parquet write timestamp int96 type. (2/2) (
eb69f5a 
#61832) (#61847)

bp #61832
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hubgeter @Thearas @morningman @doris-robot @hello-stephen @yiguolei